Intelligent automatic selection of a prior comparison study

ABSTRACT

Systems and methods for selecting a prior comparison study. One system includes an electronic processor configured to, for a medical image study associated with a patient, select a prior comparison image study. The electronic processor is also configured to automatically determine, based on monitored user interaction with the selected prior comparison image study, a usefulness of the selected prior comparison image study. The electronic processor is also configured to automatically update a selection model based on the usefulness of the prior comparison image study to a user.

FIELD

Embodiments described herein relate to selecting a prior comparisonstudy, and, more particularly, to intelligent automatic selection of aprior comparison study.

SUMMARY

A picture archiving and communication system (“PACS”) is a medicalimaging system that provides storage and access to images from multiplemodalities. In many healthcare environments, electronic images andreports are transmitted digitally via a PACS, thus eliminating the needto manually file, retrieve, or transport film jackets. A standard formatfor PACS image storage and transfer is digital imaging andcommunications in medicine (“DICOM”). Non-image data, such as scanneddocuments, may be incorporated using various standard formats such asportable document format (“PDF”) encapsulated in DICOM.

When reading a current imaging study, the reader, often a radiologistbut sometimes an orthopedic surgeon or other specialist, may obtainimportant context by seeing relevant prior comparison exams. Forexample, a wrist or an elbow would be relevant to a forearm. A chestx-ray or MR may be relevant to a shoulder image. Additionally, whenreading prior comparison image studies, the reader is generally tryingto answer questions related to whether a feature (for example, afracture, tumor, cyst, or stenosis) is new, whether a feature haschanged (for example, growth of a tumor, shrinkage of a tumor, healingof a fracture, or an increase in plaque, stenosis, aneurysm, effusion,or atrophy), or are there additional features (for example, metastasesor stenosis).

With PACS and vendor neutral archive (“VNA”) systems there is a need fora mechanism to move prior comparison imaging studies from one system (orlocation) to another system (or location) based on a triggering event,such as an order for a new study for an existing patient. Priorcomparison imaging studies may be moved from a VNA system to a PACSsystem, from a PACS system to another PACS system, from a PACS system toa viewer, and the like. Conventionally, prior imaging studies to movemay be selected by applying simple rules, such as based on matching astudy to a modality (for example, CT, MR, or DX), a body part, a timeframe (for example, most recent three years), and the like. Such rulesmay be used individually or in combination. Such a rules-based approachdoes not work well for patients having a large number of priorcomparison imaging studies (for example, more than 100 and, in somecases, more than 1000). Moving such a large number of prior comparisonimaging studies is infeasible. Additionally, existing prior relevancyrules are configured or hard coded into the PACS system or viewer, whichmakes them difficult to change or customize and resources are wastedwhen such rules are configured too broad or too narrow.

For example, when the relevancy rules are too narrow, the priorcomparison study that is most appropriate may not be available to thereading physician. However, when the relevancy rules are set toobroadly, the amount of data (the number of prior comparison studies) tobe transferred may run into other bottlenecks (for example, bandwidthand processing power) that may bog down the entire system or, due to alack of prioritization of the retrievals, hinder the availability of themost relevant study, as there are many (sometimes hundreds) of studiesqueued ahead of it for transfer. Wide area network (“WAN”) bandwidth isgenerally not unlimited and notably outbound/upload capability from agiven site may be substantially less than the download/inbound bandwidthavailable. This becomes more acute with reading physicians being morespecialized and dispersed as the reading physicians are commonly readingfor a variety of institutions, not just the studies that were performedat the same site that the reading physicians are reading at.

Other factors that exacerbate the problem of existing systems includethe late dispatch of orders, the increasing size of studies (forexample, a 70 MB 2D mammogram versus a 2-3 GB 3D mammogram), and thedrive to ever thinner slices for CT scans and MR scans that is doublingthe study sizes. Accordingly, for every factor that seems to offermitigation of the problems (more bandwidth), imaging technology advancesexacerbate the problems by almost exponentially increasing the amount ofdata to transfer.

Therefore, there is a need to for an improved selection approach toselecting studies to prefetch (for example, move to a second system orlocation for viewing) based on, for example, a new set of parameters,such as a patient study, history information, physician usage patterns,or a combination thereof.

To address these and other problems, embodiments described herein usepatterns of selection and use of prior comparison studies by a readingphysician to create new relevancy relationships. Such relevancyrelationships may be used to drive one or more of prefetching,precaching, initial selection and display of a most appropriate(relevant) prior comparison study (with or without a hanging protocol),or a combination thereof.

Accordingly, embodiments described herein provide systems and methodsfor selecting a prior comparison study. For example, one embodimentprovides a system for selecting a prior comparison image. The systemincludes an electronic processor configured to, for a medical imagestudy associated with a patient, select a prior comparison image study.The electronic processor is also configured to automatically determine,based on monitored user interaction with the selected prior comparisonimage study, a usefulness of the selected prior comparison image study.The electronic processor is also configured to automatically update aselection model based on the usefulness of the prior comparison imagestudy to a user.

Another embodiment provides a method of selecting a prior comparisonimage. The method includes, for a medical image study associated with apatient, selecting, with an electronic processor, a prior comparisonimage study. The method also includes automatically determining, withthe electronic processor, based on monitored user interaction with theselected prior comparison image study, a usefulness of the selectedprior comparison image study. The method also includes automaticallyupdating, with the electronic processor, a selection model based on theusefulness of the prior comparison image study to a user.

Yet another embodiment provides a non-transitory computer readablemedium including instructions that, when executed by an electronicprocessor, causes the electronic processor to execute a set offunctions. The set of functions includes, for a medical image studyassociated with a patient, selecting a prior comparison image study. Theset of functions also includes automatically determining based onmonitored user interaction with the selected prior comparison imagestudy, a usefulness of the selected prior comparison image study. Theset of functions also includes automatically updating a selection modelbased on the usefulness of the prior comparison image study to a user.

Other aspects of the embodiments will become apparent by considerationof the detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for selecting a prior comparison studyaccording to some embodiments.

FIG. 2 illustrates a server included in the system of FIG. 1 accordingto some embodiments.

FIG. 3 is a flowchart illustrating a method for selecting a priorcomparison study using the system of FIG. 1 according to someembodiments.

DETAILED DESCRIPTION

One or more embodiments are described and illustrated in the followingdescription and accompanying drawings. These embodiments are not limitedto the specific details provided herein and may be modified in variousways. Furthermore, other embodiments may exist that are not describedherein. Also, the functionality described herein as being performed byone component may be performed by multiple components in a distributedmanner. Likewise, functionality performed by multiple components may beconsolidated and performed by a single component. Similarly, a componentdescribed as performing particular functionality may also performadditional functionality not described herein. For example, a device orstructure that is “configured” in a certain way is configured in atleast that way, but may also be configured in ways that are not listed.Furthermore, some embodiments described herein may include one or moreelectronic processors configured to perform the described functionalityby executing instructions stored in non-transitory, computer-readablemedium. Similarly, embodiments described herein may be implemented asnon-transitory, computer-readable medium storing instructions executableby one or more electronic processors to perform the describedfunctionality. As used in the present application, “non-transitorycomputer-readable medium” comprises all computer-readable media but doesnot consist of a transitory, propagating signal. Accordingly,non-transitory computer-readable medium may include, for example, a harddisk, a CD-ROM, an optical storage device, a magnetic storage device, aROM (Read Only Memory), a RAM (Random Access Memory), register memory, aprocessor cache, or any combination thereof.

In addition, the phraseology and terminology used herein is for thepurpose of description and should not be regarded as limiting. Forexample, the use of “including,” “containing,” “comprising,” “having,”and variations thereof herein is meant to encompass the items listedthereafter and equivalents thereof as well as additional items. Theterms “connected” and “coupled” are used broadly and encompass bothdirect and indirect connecting and coupling. Further, “connected” and“coupled” are not restricted to physical or mechanical connections orcouplings and can include electrical connections or couplings, whetherdirect or indirect. In addition, electronic communications andnotifications may be performed using wired connections, wirelessconnections, or a combination thereof and may be transmitted directly orthrough one or more intermediary devices over various types of networks,communication channels, and connections. Moreover, relational terms suchas first and second, top and bottom, and the like may be used hereinsolely to distinguish one entity or action from another entity or actionwithout necessarily requiring or implying any actual such relationshipor order between such entities or actions.

As discussed above, when reading a current imaging study, the reader,often a radiologist but sometimes an orthopedic surgeon or otherspecialist, may obtain important context by seeing relevant priorcomparison exams. However, existing rules used to obtain or retrieveprior comparison imaging studies suffer from numerous deficiencies, asdescribed above. Therefore, there is a need to for an improved selectionapproach to selecting studies to prefetch (for example, move to a secondsystem or location for viewing) based on, for example, a new set ofparameters, such as a patient study, history information, physicianusage patterns, or a combination thereof.

To address these and other problems, embodiments described herein usepatterns of selection and use of prior comparison studies by a readingphysician to create new relevancy relationships. Such relevancyrelationships may be used to drive one or more of prefetching,precaching, initial selection and display of a most appropriate(relevant) prior comparison study (with or without a hanging protocol),or a combination thereof.

FIG. 1 schematically illustrates a system 100 for selecting a priorcomparison study (for example, a prior comparison image or set ofimages) according to some embodiments. The system 100 includes a server105, an archive 115, a plurality of user devices 117 (referred to hereincollectively as “the user devices 117” and individually as “the userdevice 117”), and a plurality of imaging modalities 130 (referred toherein collectively as “the imaging modalities 130” and individually as“the imaging modality 130”). In some embodiments, the system 100includes fewer, additional, or different components than illustrated inFIG. 1. For example, the system 100 may include multiple servers 105,archives 115, or a combination thereof.

The server 105, the archive 115, the user devices 117, and the imagingmodalities 130 communicate over one or more wired or wirelesscommunication networks 140. Portions of the communication network 140may be implemented using a wide area network, such as the Internet, alocal area network, such as a Bluetooth™ network or Wi-Fi, andcombinations or derivatives thereof. Alternatively or in addition, insome embodiments, components of the system 100 communicate directly ascompared to through the communication network 140. Also, in someembodiments, the components of the system 100 communicate through one ormore intermediary devices not illustrated in FIG. 1.

The server 105 is a computing device, which may serve as a gateway forthe archive 115. For example, in some embodiments, the server 105 may bea PACS server. Alternatively, in some embodiments, the server 105 may bea server that communicates with a PACS server to access the archive 115.As illustrated in FIG. 2, the server 105 includes an electronicprocessor 150, a memory 155, and a communication interface 160. Theelectronic processor 150, the memory 155, and the communicationinterface 160 communicate wirelessly, over one or more communicationlines or buses, or a combination thereof. The server 105 may includeadditional components than those illustrated in FIG. 2 in variousconfigurations. The server 105 may also perform additional functionalityother than the functionality described herein. Also, the functionalitydescribed herein as being performed by the server 105 may be distributedamong multiple devices, such as multiple servers included in a cloudservice environment. In addition, in some embodiments, the user devices117 may be configured to perform all or a portion of the functionalitydescribed herein as being performed by the server 105.

The electronic processor 150 includes a microprocessor, anapplication-specific integrated circuit (ASIC), or another suitableelectronic device for processing data. The memory 155 includes anon-transitory computer-readable medium, such as read-only memory(“ROM”), random access memory (“RAM”) (for example, dynamic RAM(“DRAM”), synchronous DRAM (“SDRAM”), and the like), electricallyerasable programmable read-only memory (“EEPROM”), flash memory, a harddisk, a secure digital (“SD”) card, another suitable memory device, or acombination thereof. The electronic processor 150 is configured toaccess and execute computer-readable instructions (“software”) stored inthe memory 155. The software may include firmware, one or moreapplications, program data, filters, rules, one or more program modules,and other executable instructions. For example, the software may includeinstructions and associated data for performing a set of functions,including the methods described herein.

For example, as illustrated in FIG. 2, the memory 155 may store alearning engine 165 and a model database 170. In some embodiments, thelearning engine 165 develops a model using one or more machine learningfunctions. Machine learning functions are generally functions that allowa computer application to learn without being explicitly programmed. Inparticular, a computer application performing machine learning functions(sometimes referred to as a learning engine) is configured to develop analgorithm based on training data. For example, to perform supervisedlearning, the training data includes example inputs and correspondingdesired (for example, actual) outputs, and the learning engineprogressively develops a model that maps inputs to the outputs includedin the training data. Machine learning may be performed using varioustypes of methods and mechanisms including but not limited to decisiontree learning, association rule learning, artificial neural networks,inductive logic programming, support vector machines, clustering,Bayesian networks, reinforcement learning, representation learning,similarity and metric learning, sparse dictionary learning, and geneticalgorithms.

Using all of these approaches, a computer program may ingest, parse, andunderstand data and progressively refine models for data analytics,including image analytics.

Models generated by the learning engine 165 may be stored in the modeldatabase 170. As illustrated in FIG. 2, the model database 170 isincluded in the memory 155 of the server 105. It should be understoodthat, in some embodiments, the model database 170 is included in aseparate device accessible by the server 105 (included in the server 105or external to the server 105).

The communication interface 160 allows the server 105 to communicatewith devices external to the server 105. For example, as illustrated inFIG. 1, the server 105 may communicate with the archive 115, the userdevices 117, the imaging modalities 130, or a combination thereofthrough the communication interface 160. In particular, thecommunication interface 160 may include a port for receiving a wiredconnection to an external device (for example, a universal serial bus(“USB”) cable and the like), a transceiver for establishing a wirelessconnection to an external device (for example, over one or morecommunication networks 140, such as the Internet, local area network(“LAN”), a wide area network (“WAN”), and the like), or a combinationthereof.

The user device 117 is also a computing device and may include a desktopcomputer, a terminal, a workstation, a laptop computer, a tabletcomputer, a smart watch or other wearable, a smart television orwhiteboard, or the like. Although not illustrated, the user device 117may include similar components as the server 105 (an electronicprocessor, a memory, and a communication interface). The user device 117may also include a human-machine interface for interacting with a user.The human-machine interface may include one or more input devices, oneor more output devices, or a combination thereof. Accordingly, in someembodiments, the human-machine interface allows a user to interact with(for example, provide input to and receive output from) the user device117. For example, the human-machine interface may include a keyboard, acursor-control device (for example, a mouse), a touch screen, a scrollball, a mechanical button, a display device (for example, a liquidcrystal display (“LCD”)), a printer, a speaker, a microphone, or acombination thereof. In some embodiments, the human-machine interfaceincludes a display device. The display device may be included in thesame housing as the user device 117 or may communicate with the userdevice 117 over one or more wired or wireless connections. For example,in some embodiments, the display device is a touchscreen included in alaptop computer or a tablet computer. In other embodiments, the displaydevice is a monitor, a television, or a projector coupled to a terminal,desktop computer, or the like via one or more cables. Some of thesedevices may have severe memory or computational capacity constraintsplacing an even greater emphasis on intelligent prefetching of a studyor a study element.

The imaging modalities 130 provide imagery (for example, medicalimages). The imaging modalities 130 may include a computed tomography(CT), a magnetic resonance imaging (MRI), an ultrasound (US), anothertype of imaging modality, or a combination thereof. While theembodiments described herein are generally described in the context ofradiology or cardiology medical images, it should be understood thatother images, such as pathology images, including gross specimen photos,microscopy slide images, and whole scanned slide datasets, may also beused. Other images, such as dermatology, intra-operative or surgery, orwound care photos or movies, may also be used. In some embodiments, themedical images are transmitted from the imaging modality 130 to a PACSGateway (for example, the server 105), before being stored in thearchive 115.

The archive 115 stores a plurality of medical images (referred to hereincollectively as “the medical images” and individually as “the medicalimage”). Accordingly, the archive 115 provides for the storage andretrieval of images and reports. In some embodiments, the archive 115 iscombined with the server 105. Alternatively or in addition, the medicalimages may be stored within a plurality of databases, such as within acloud service. Although not illustrated in FIG. 1, the archive 115 mayinclude components similar to the server 105, such as an electronicprocessor, a memory, a communication interface and the like. Forexample, the archive 115 may include a communication interfaceconfigured to communicate (for example, receive data and transmit data)over the communication network 140.

A user may use the user device 117 to access and view medical images.Accordingly, in some embodiments, the user devices 117 are workstationsfor interpreting and reviewing medical images (for example, one or moreimages) stored in the archive 112. For example, a radiologist may usethe user device 117 as a viewing or reading workstation to review apatient's study (a current or a new study or one or more priorcomparison studies) and formulate a diagnosis for the patient. In someembodiments, a secured network is used for the transmission of patientinformation between the components of the system 100 (for example, thecommunication network 140).

For example, in some embodiments, the electronic processor 150 receivesthe medical image as a new medical image associated with a patient fromone of the imaging modalities 130 (via the communication network 140).For example, when a new medical imaging study is ordered for a patientand captured by one of the imaging modalities 130, the new medical imageis transmitted from the imaging modality 130 to the electronic processor150. Alternatively or in addition, in some embodiments, the electronicprocessor 150 receives the medical image from the archive 115 inresponse to a request for the medical image from a user. For example, auser may use the user device 117 to initiate a request for a particularmedical image associated with a patient. In response to receiving therequest for the particular medical image, the electronic processor 150receives the medical image from the archive 115. Accordingly, in someembodiments, the medical image received by the electronic processor 150is a prior comparison image and is included in the plurality of priorcomparison images stored in the archive 115.

In some embodiments, the user devices 117 are web-based viewers. Forexample, the user may access the medical images from the archive 115(through a browser application or a dedicated application stored on theuser device 117 that communicates with the server 105) and view themedical images on the display device associated with the user device117. Accordingly, in some embodiments, the user devices 117 provideweb-based interfaces. Such web-based interfaces may be accessed via theInternet or a wide area network (“WAN”). In some embodiments, connectionsecurity is provided by a virtual private network (“VPN”), a securesockets layer (“SSL”), or a combination thereof. A client's sidesoftware (at a user device 117) may comprise ActiveX, JavaScript, a JavaApplet, or an iOS or Android application. PACS clients may also be fullapplications, which utilize the full resources of the computer that thePACS clients are executing on outside of a web environment.

As noted above, the quantity of available prior comparison studies for agiven patient and the limitations associated with existing rule-basedselection approaches interfere with a reviewer's ability to efficientlyaccess and view prior comparison studies associated with a patient, and,ultimately, with the reviewer's ability to formulate a diagnosis for thepatient. To solve this and other problems, the system 100 is configuredto automatically select (prefetch) prior comparison studies (forexample, a first medical image, a second medical image, and the like).Based on the selection of the prior comparison studies, the methods andsystems described herein display the selected prior comparison studiesto a user (for example, a reviewer).

For example, FIG. 3 is a flowchart illustrating a method 300 forselecting a prior comparison study (for example, a medical image or setof images) according to some embodiments. The method 300 is describedhere as being performed by the server 105 (the electronic processor 150executing instructions). However, as noted above, the functionalityperformed by the server 105 (or a portion thereof) may be performed byother devices, including, for example, the user device 117 (via anelectronic processor executing instructions).

As illustrated in FIG. 3, the method 300 includes, for a medical imagestudy associated with a patient, selecting, with the electronicprocessor 150, a prior comparison image study (at block 305). In someembodiments, the electronic processor 150 automatically selects theprior comparison image study using a selection model. Alternatively orin addition, in some embodiments, the electronic processor 150automatically selects the prior comparison image study based on one ormore selection rules. The selection rule may be a predetermined (preset)selection rule or a dynamic selection rule that is modified based onmonitored user interactions, extracted attributes, or a combinationthereof (as described in greater detail below). Similarly, as alsodescribed below, the selection model may be updated with feedback. Asdescribed below, in some embodiments, the electronic processor 150 mayinitially use one or more selection rules to select prior comparisonimage studies but may disable the one or more of the selection rules anduse the selection model to automatically select the prior comparisonimage study as the selection model is trained using feedback on theimage studies selected via the rules, the model, or both. The selectionmodel may be built using various factors, including the factorsdescribed below. For example, training information may be obtained forhistorically-selected prior comparison image studies and used togenerate the selection model using one or more machine learningtechniques (e.g., via the learning engine 165). Accordingly, the factorsdiscussed below may be used as part of training a selection model, aspart of establishing a selection rule, or both.

As noted above, when reading a current imaging study, the reader mayobtain important context by seeing relevant prior comparison exams.Accordingly, the selected prior comparison image study may include oneor more prior comparison images that may be relevant to, for example,obtaining a context associated with the medical image study, the patientassociated with the medical image, or a combination thereof. In otherwords, the selected prior comparison image study may help a readeranswer questions related to whether a feature is new, whether a featurehas changed, or whether there are additional features. The electronicprocessor 150 may automatically select the prior comparison image studyas part of performing pre-fetching, pre-caching, or selecting an imagestudy for initial display. Accordingly, in some embodiments, theelectronic processor 150 transmits the prior comparison image study to auser for display via a display device of the user device 117.

In some embodiments, the electronic processor 150 selects a priorcomparison image study by analyzing the medical image associated withthe patient to extract one or more attributes from the medical image.The electronic processor 150 may select the prior comparison image studybased on the one or more attributes extracted from the medical image.Alternatively or in addition, the electronic processor 150 selects aprior comparison image study by analyzing a medical image associatedwith a patient to extract one or more attributes from the medical image(a first set of attributes), analyzing a prior comparison image study toextract one or more attributes associated with the prior comparisonimage (a second set of attributes), and comparing the one or moreattributes extracted from the medical image with the one or moreattributes associated with the prior comparison image study (forexample, performing a comparison of a first set of attributes with asecond set of attributes). The electronic processor 150 may select aprior comparison image study based on the comparison. Alternatively orin addition, the electronic processor 150 selects a prior comparisonimage study by analyzing a plurality of prior comparison image studiesto extract one or more attributes associated with each prior comparisonimage. The electronic processor 150 may then select a prior comparisonimage study based on the one or more attributes associated with eachprior comparison image.

In some embodiments, the one or more attributes extracted from a currentmedical image or one or more of the plurality of prior comparison imagestudies may be extracted by the electronic processor 150 using naturallanguage processing. Alternatively or in addition, the electronicprocessor 150 extracts the one or more attributes using ontologicalmapping of clinical terms or findings within an order, a report visitnote, or another document associated with the medical image.

There are a number of features (or attributes) of a current medicalimage and a prior comparison image study that may be used in determininga usefulness (relevancy). A source of the extracted attributes mayinclude the imaging study metadata (for example, a DICOM header). Themetadata may include a variety of text (for example, a studydescription, a reason for exam, a body part examined, a laterality, andthe like), codified data (for example, ICD 9/10 codes), a body part, aview descriptor (for example, CC ML, MLO, and other views of a breast),use of a contrast agent, a method of administration, a seriesdescription, an acquisition parameter (for example, a spin, an echotime, a relaxation time, a flip angle in an MR study) and the like.Additionally, a delta of a date of a prior comparison study and acurrent study may also be computed (for example, via the electronicprocessor 150).

Another source of the extracted attributes may include a Health-Level 7(“HL7”) Order. HL7 refers to a set of international standards fortransfer of clinical and administrative data between softwareapplications used by various healthcare provides. A HL7 Order is linkedto a medical image (a medical image study) that may be extracted fromthe HL7 ORM (Order Message), which contains additional information thatis not transferred to the image metadata. The additional informationextracted from the HL7 ORM may include, for example, items such as a CPTcode, a reason for an exam (much larger than may be accommodated in aDICOM header), a procedure step, and patient history questionnaire datathat may have been gathered by a technologist performing the exam.

Yet another source of the extracted attributes may include a physician'sreport for a prior comparison image study. A physician's report may beprocessed using natural language processing to extract key attributes,such as a history, an indication, a finding, an impression, aconclusion, and the like. One or more of the extracted attributes may bemapped to an ontology, such as UMLS, as is done in Watson Health ImagingPatient Synopsis and Clinical Review to then allow for subsequentcomparison based on standardized terminology.

A further source of the extracted attributes may include a most recentvisit or clinical note for a patient by an ordering physician. The mostrecent visit or clinical note may be processed (with the electronicprocessor 150) using natural language processing to extract the one ormore attributes from the clinical content of the most recent visit orclinical note. The most recent visit or clinical notes may contain moreclinical context than a notoriously vague order (for example, pneumoniasuspected based on patient cough and presentation, but simply order a “2view chest x-ray” with no further information in the order).

After selecting the prior comparison image study (and transmitting theselected prior comparison image study to a user for display via adisplay device of the user device 117), the electronic processor 150determines a usefulness of the prior comparison image study (at block310). In some embodiments, the electronic processor 150 automaticallydetermines the usefulness of the selected prior comparison image study.For example, in some embodiments, the electronic processor 150automatically determines the usefulness of the selected prior comparisonimage study based on monitored user interaction with the selected priorcomparison image study (considered alone or in combination with otherfactors, such as attributes of the selected image studies, reportsassociated with the selected images studies, or the like as describedbelow).

As noted above, the electronic processor 150 may determine theusefulness of the selected prior comparison image study based onmonitored user interactions with the selected prior comparison imagestudy. The monitored user interactions may include, for example, one ormore interactions of the user with the medical image, one or more priorcomparison images, or a combination thereof. In particular, themonitored user interactions may include, for example, (a) a measurementof how long a prior comparison study is kept opened, (b) a lastcomparison study viewed (presumably answering a clinical question), (c)a first comparison opened with no additional selection (and no similartypes of comparisons available), (d) a first comparison study opened(for example, many mammograph readers prefer to view the two or threeyear old prior comparison study presented first for comparison beforereviewing some or all of the prior comparison studies), (e) textanalytics finding a simple mention in a report of a prior comparisonstudy (some PACS will note ANY study opened for comparison in the reportautomatically whether it was useful or not), (f) text analytics of areport of changes of an anatomical structure, a disease, a structuraldefect or neoplasm (tumor) where these may be qualitative ofquantitative comparisons, (g) opening and discarded prior comparisonimages may still be valuable, but discarded or replaced comparisons maybe ranked lower than a last or report mentioned prior comparison study,(h) retrievals to the PACS from the VNA (this may be confounded byautomated precaching when used for one or more types of studies), (i)retrievals to the PACS of a prior comparison study that is separated intime from the rest of the prior comparison study retrievals(differentiate between automated precaching and manual selection), (j)explicit manual selection of a prior comparison study or series, and thelike.

Accordingly, in some embodiments, the electronic processor 150automatically determines the usefulness of the selected prior comparisonimage study by considering how long the prior comparison image study isopened, whether the prior comparison image study was the last studyviewed before completing a report for the medical image study associatedwith the patient, or a combination thereof. Alternatively or inaddition, the electronic processor 150 may automatically determine theusefulness of the selected prior comparison image study by determiningwhether the prior comparison image study was a first image study openedby the user, determining whether, using natural language processing of areport generated for the medical image study associated with thepatient, references the prior comparison image study, or a combinationthereof. Alternatively or in addition, the electronic processor 150 mayautomatically determine the usefulness of the selected prior comparisonimage study by identifying whether another prior comparison image studywas retrieved for the medical image study associated with the patient,comparing the prior comparison image study with another prior comparisonimage study retrieved for the medical image study associated with thepatient, or a combination thereof. In some embodiments, the electronicprocessor 150 automatically determines a usefulness of the selectedprior comparison image by determining whether, using natural languageprocessing of a report generated for the medical image study associatedwith the patient, references the prior comparison image study. In someembodiments, electronic processor 150 automatically determines theusefulness of the selected prior comparison image by performing naturallanguage processing of a report generated for the medical image studyassociated with the patient.

Returning to FIG. 3, the method 300 also includes updating, with theelectronic processor 150, a selection model (at block 315). In someembodiments, the selection model is generated using machine learningbased on historical patterns of selection and use of prior comparisonstudies for a plurality of medical image studies. In some embodiments,the electronic processor 150 updates the selection model automatically.In some embodiments, the electronic processor 150 updates the selectionmodel based on the usefulness of the prior comparison image study to auser.

In some embodiments, the electronic processor 150 selects the priorcomparison image study based on a previously-determined usefulness ofthe prior comparison image study. For example, the electronic processor150 may select a particular prior comparison image study from aplurality of prior comparison image studies based on the usefulness ofthat particular prior comparison image study exceeding (or satisfying) apredetermined usefulness threshold. Alternatively or in addition, thepredetermined usefulness threshold may define a total number of priorcomparison images to include in the prior comparison image study. Forexample, when the predetermined usefulness threshold is five, theelectronic processor 150 identifies the five prior comparison imageshaving the greatest usefulness for inclusion in the selected priorcomparison image study.

In some embodiments, the electronic processor 150 also determines arelevancy order for displaying each prior comparison image included inthe selected prior comparison image study. The relevancy order may bebased on the usefulness of each prior comparison image included in theselected prior comparison image study. In other words, the relevancyorder may reflect the usefulness associated with each prior comparisonimage included in the selected prior comparison image study. Forexample, a first prior comparison image included in the selected priorcomparison image study having the greatest usefulness may be in a firstposition of the relevancy order while a second prior comparison imageincluded in the selected prior comparison image study having a lowerusefulness than the first prior comparison image may be in a secondposition of the relevancy order (the next position of the relevancyorder after the first position). Accordingly, the usefulness may be usednot just to drive prefetching or precaching of prior comparison studies,but also a display order of the prior comparisons within a PACS viewer,replacing the traditional hanging protocol prior relevancy rules orselection process, such that the most appropriate prior comparison studyis loaded automatically and the next most relevant prior comparisonstudies are either available for manual selection, concurrent display,or sequential display.

In some embodiments, the monitored user interactions may include arequest for an additional prior comparison image. For example, theelectronic processor 150 may receive a request for an additional priorcomparison image from the user. In response to receiving the request forthe additional prior comparison image, the electronic processor 150 maytransmit the additional prior comparison image to the user for display.In some embodiments, the electronic processor 150 uses the request foran additional prior comparison image from the user as feedback (trainingdata) for selecting prior comparison images (using the selection modeldeveloped with machine learning). In particular, the electronicprocessor 150 may update the selection model based on the request forthe additional prior comparison image. In some embodiments, theelectronic processor 150 automatically selects a new prior comparisonimage study using the updated selection model and transmits the newprior comparison image study to a user for display.

Accordingly, with the “extracted attribute sets” available for thecurrent medical image, one or more prior comparison images, or acombination thereof, the system 100 may not only be preconfigured ortrained to provide an appropriate set of prior relevancy criteria (forexample, the selected prior comparison image study) based on the broadset of extracted attributes from the sources mentioned above, but alsolearn from the actions (monitored user interactions) of the readingphysician either individually/personally or in aggregate (for example,to update the selection model). Alternatively or in addition, in someembodiments, the electronic processor 150 uses one or more selectionrules as a means to prime the selection process until intelligentselection is trained.

With the extracted attributes available for comparison and determiningusefulness (relevancy), the system 100 is able to learn betterassociations (between a current medical image and one or more priorcomparison studies) that may be applied by simple rules (selection rulesor modified selection rules based on monitored user interactions). Asnoted above, user actions may be monitored as well as natural languageprocessing extraction from a current medical image and related documents(for example, an order, a visit note, a report, and the like) to thenprovide the system 100 the ability to learn prior relevancy patterns(monitored user interactions) and update the prior relevancy patterns ina continuous learning process. In some embodiments, usage patterns andresulting imaging report content may replace a process of manuallyannotating training data sets with a continuous feedback process thatmay either be purely implicit with no overt user actions in training thesystem 100 (for example, one or more models) or explicit where a usermay indicate that a suggested prior comparison image has either high orlow relevancy (usefulness) and why. In other words, by applying themonitored user interactions, extracted attributes, or a combinationthereof described herein along with a user indication of a closeness ofmatch or best prior relevancy as a corrective or annotative input to thelearning process, where the extracted feature sets (attributes) may bethe original data input into the learning process, the system 100 (forexample, one or more models) may be trained to better select priorcomparison studies and rank the prior comparison studies in an order inwhich the prior comparison studies may be retrieved.

Embodiments described herein may be either deeply integrated into a PACSviewer or alternatively used on a VNA or PACS server (for example, theserver 105), where all that may be seen are the retrievals by a vieweror PACS client. When applying the embodiments described herein toexisting PACS or VNA, not all of the feature set information may beavailable (for example, no visit or clinical notes available in an HL7feed to the PACS). Additionally, there may only be the automatedinferential feedback mechanisms (especially when dealing with thirdparty PACS or viewers) and this is deployed as a feature of the VNA.Accordingly, in some embodiments, the methods and systems describedherein provide a means of discriminating between retrievals that aredriven by existing hanging protocols and relevancy rules in a viewerversus what was actually found to be most useful or relevant by a user.

Some PACS or viewers automatically retrieve a few of the most relevantprior comparison images as a background process in advance of a useropening a study for viewing. This is often referred to as “precaching”of the study. However, not all prior comparison studies or imagesselected for viewing will be good or optimal choices, whether this isdone algorithmically with a hanging protocol or relevancy rules or donemanually by a user opening a study or selecting images from the studyfor comparison. Therefore, a simple “was a comparison study viewed?”tracking is insufficient.

Thus, the embodiments herein provide, among other things, methods andsystems that utilize actions of a user (whether directly or indirectly)to provide better prior relevancy ranking for prior comparison studies.For example, knowing which prior comparison studies were recalled ordisplayed, the sequencing of the prior comparison studies, a dwell orviewing time, as well as whether or not an already displayed priorcomparison study was replaced, may be utilized along with metadata of astudy to train the systems described herein. The metadata may come fromthe DICOM images of a current medical image, a prior comparison image, aHL7 order, a report associated with a prior comparison study, anothertype of metadata, or a combination thereof. Accordingly, the embodimentsdescribed herein prefetch and display the most relevant prior comparisonstudies as well as prioritize a transfer or retrieval of the next mostrelevant prior comparison studies.

Therefore, the embodiments herein provide, among other things, automatedlearning of a relevancy of prior comparison study based on user actions,such as an order and a timing of viewing of prior comparison studies andtheir reports, analyzed against metadata for an imaging study, an order,and a report for the prior comparison studies. User actions where priorcomparison studies replace existing prior comparison studies in a PACSor another viewer and a sequence of replacement drives new learning of aprioritization ranking increase or decrease of the prior comparisonstudy.

Various features and advantages of the invention are set forth in thefollowing claims.

What is claimed is:
 1. A system for prefetching or precaching a priorcomparison image, the system comprising: an electronic processorconfigured to, in response to receiving a new medical image study for anexisting patient: automatically determine a usefulness of a priorcomparison image study to the new medical image study by consideringwhether the prior comparison image study was a last medical mage studyviewed before completing a prior report associated with the existingpatient, and automatically prefetch or precache the prior comparisonimage study based on the determined usefulness of the prior comparisonimage study to the new medical image study being above a threshold. 2.The system of claim 1, wherein the electronic processor is configured toautomatically select the prior comparison image study based on aselection rule.
 3. The system of claim 2, wherein the electronicprocessor is configured to disable the selection rule and automaticallyselect the prior comparison image study using the selection model. 4.The system of claim 1, wherein the electronic processor is configured toautomatically select the prior comparison image study using a selectionmodel based on the usefulness of the prior comparison image study to thenew medical image study.
 5. The system of claim 1, wherein theelectronic processor is configured to automatically determine theusefulness of the prior comparison image study to the new medical imagestudy by considering how long the prior comparison image study wasopened.
 6. The system of claim 1, wherein the electronic processor isconfigured to automatically determine the usefulness of the priorcomparison image study to the new medical image study by determiningwhether the prior comparison image study was a first image study openedby the user.
 7. The system of claim 1, wherein the electronic processoris configured to automatically determine the usefulness of the priorcomparison image study to the new medical image study by determiningwhether, using natural language processing of the prior reportassociated with the existing patient, references the prior comparisonimage study.
 8. The system of claim 1, wherein the electronic processoris configured to automatically determine the usefulness of the priorcomparison image study to the new medical image study by performingnatural language processing of the prior report associated with theexisting patient.
 9. The system of claim 1, wherein the electronicprocessor is configured to automatically determine the usefulness of theprior comparison image study to the new medical image study byidentifying whether another prior comparison image study was retrievedfor the prior report associated with the existing patient.
 10. Thesystem of claim 9, wherein the electronic processor is configured toautomatically determine the usefulness of the prior comparison imagestudy to the new medical image study by comparing the prior comparisonimage study with another prior comparison image study retrieved for theprior report associated with the existing patient.
 11. A method forprefetching or precaching a prior comparison image, the methodcomprising: in response to receiving a new medical image study for anexisting patient: automatically determining, with an electronicprocessor, a usefulness of a prior comparison image study to the newmedical image study by considering whether the prior comparison imagestudy was a last study viewed before completing a prior reportassociated with the existing patient; and automatically prefetching orprecaching the prior comparison image study based on the determinedusefulness of the prior comparison image study to the new medical imagestudy being above a threshold.
 12. The method of claim 11, furthercomprising automatically updating a selection model generated usingmachine learning based on historical patterns of selection and use ofprior comparison studies for a plurality of medical image studies. 13.The method of claim 11, further comprising automatically selecting theprior comparison image study based on a selection rule.
 14. The methodof claim 11, further comprising automatically selecting the priorcomparison image study using a selection model.
 15. A non-transitorycomputer readable medium including instructions that, when executed byan electronic processor, causes the electronic processor to execute aset of functions, the set of functions comprising: in response toreceiving a new medical image study for an existing patient:automatically determining, with an electronic processor, a usefulness ofa prior comparison image study to the new medical image study byconsidering whether the prior comparison image study was a last studyviewed before completing a prior report associated with the existingpatient; and automatically prefetching or precaching the priorcomparison image study based on the determined usefulness of the priorcomparison image study to the new medical image study being above athreshold.
 16. The non-transitory computer readable medium of claim 15,wherein the automatically determining the usefulness of the priorcomparison image study to the new medical image study includesautomatically determining the usefulness of the prior comparison imagestudy to the new medical image study by determining whether the priorcomparison image study was a first image study opened by a user.
 17. Thenon-transitory computer readable medium of claim 15, wherein the set offunctions further includes automatically selecting the prior comparisonimage study using a selection model.